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OUTLINE 


•  INTRODUCTION  (Melese) 

•  THEORETICAL  FOUNDATIONS  (Melese) 


•  THE  NATIONAL  ACADEMY  OF  SCIENCE 
SURVEY  OF  AIR  FORCE  PROGRAM 
MANAGERS  (Dillard) 


•  OUR  PRELIMINARY  ANALYSIS  OF  THE 


DATA  (Franck) 

•  CONCLUDING  REMARKS 
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Acquisition  Challenge: 
Performance,  Cost  and  Schedule 


•  Weapons  acquisition  is  afflicted  by  similar 
pathologies  that  impact  public  sector 
procurement: 

-  Conflicting  goals  &  objectives 

-  Information  asymmetry, 

-  Lack  of  credible  commitments 


•  In  addition  to 


-  Technical  complexity, 

-  Asset  specificity,  and 

-  Uncertainty  (funding:  demand;  design;  production) 
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Background 


•  2008,  National  Academy  of  Sciences  study  for 
Optimizing  U.  S.  Air  Force  and  Department  of 
Defense  Review  of  Air  Force  Acquisition 
Programs 


•  Myriad  of  programmatic  and  technical  reviews 
prescribed  for  space  and  non-space  system 
acquisition  programs 


•  Options  to  streamline,  consolidate,  reduce? 
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Contribution  of  Our  Study 

•  Examine  roles  of  monitoring  &  oversight 

-  Programmatic  and  technical  reviews 

•  Within  the  context  of  two  frameworks: 

-  Principal-Agent  Model  (Agency  Theory) 
-Transaction  Cost  Economics  (TCE) 

•  Begin  to  describe  benefits  &  costs  of  current 
monitoring  &  oversight 

-  USAF  Survey  for  National  Academy  of  Sciences 


Principal-Agent  Problem 


Principal  <- 


Agent 


Congress 

OSD 

DAB 


Programmatic 

Reviews 
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Principal-Agent  Model 

•  Model  Assumptions — Principal  and  Agents  have: 

-  Conflicting  Goals  &  Objectives  of  different  players 

•  (Congress,  OSD,  PEO,  PM,  Contractor,  etc.) 

-  Asymmetric  Information 

•  Agent  has  information  advantage  over  Principal 

-  Adverse  Selection:  Only  Agent  knows  his  type  (abilities,  costs) 

-  Moral  Hazard:  Only  Agent  knows  his  effort  (quality,  savings,...) 

-  Imperfect  Information 

•  Uncertain  correlation  between  Effort  and  Outcomes 

•  Agents  are  risk  averse 


•  Goal: 


-  Evaluate  alternative  governance  mechanisms  that  align 
interests  of  Agents  with  Principal 


Aligning  Interests  of  Agents  with  Principal 


•  Monitoring  &  Oversight  (Reagan:  “Trust  but  Verify”) 

-  Ex-ante  controls 

•  Legal  &  Administrative  Procedures 

•  Reporting  Requirements 

-  Restricts  Agents’  flexibility  (administrators  vs.  managers) 

-  Ex-post  oversight 

•  Budgetary  Restrictions  (color  of  money) 

•  Monitoring  (What?  &  How?)  or  Surveillance  of  Agent 

-  “Police  patrol”:  Continued  &  detailed  oversight  &  direct 
supervision 

-  “Fire  alarm”:  Rely  on  third  parties  (GAO;etc.)  for  oversight 

-  Regulatory  oversight  cost  premium  reduces  competition 

-  Can  properly  structured  incentives  &  other  governance  mechanisms 
substitute  for  some  monitoring  &  oversight? 
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Transaction  Costs:  Monitoring  & 
Oversight 


•  According  to  Heritage  Foundation  study 

•  For  six  months  in  the  early  1990s,  Ambassador  Henry  F. 
Cooper  (then  director  of  the  Strategiv  Defense  Initiative 
Organization),  tracked  the  costs  to  support  one  decision 
point  in  THAAD  (Theater  High  Altitude  Area  Defense).  It 
took: 


-  75,000  government  labor  hours 

-  250,000  contractor  labor  hours 

-  Over  1  ton  of  documents 

-  $22  million 


efense  Acquisition  in  Transition 

£_H  ANNUAL  ACQUISITION  RESEARCH  SYMPOSIUM 


May  12-14,  2009 
Monterey,  CA 


A  Transactions  Cost  Guide 


In  making  outsourcing  decisions,  it  is  important  to 
consider  not  only  the  internal  and  external  production 
costs  of  providing  the  good  or  service,  but  also  the 
cost  of  managing  the  transaction  internally  or 
externally. 


Total  Costs  =  Production  Costs  +  Transaction  Costs 


Transaction  Costs  =  Search  and  Information  Costs  +  Bargaining 
Decision  and  Contracting  Costs  +  Policing  and  Enforcement  Costs 
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Transaction  Cost  Economics  (TCE) 


•  Objective:  Design  governance  mechanisms  that 
minimize  coordination  costs,  and  the  costs  of 
motivating  effort  and  investments  to  achieve  “better, 
faster,  cheaper.” 


•  Main  insight  of  TCE  is  that  understanding  key 
characteristics  of  a  transaction  can  help  guide  the 
optimal  choice  of  governance  mechanisms 


-  Anticipate  re-negotiation  costs  (hold-up,  change  orders,  lock-in,  etc.) 

-  If  ex-post  opportunistic  transaction  costs  are  ignored,  then  future  cost 
increases  may  not  be  properly  anticipated. 
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Spectrum  of  Market  Structures 
(Characteristics  of  the  Product  Market) 


#  BUYERS 


many 

Monopoly 

Oligopoly 

Competitive  Market 

few 

Game  Theory 

1 

Principal- Agent  Problem 

Monopsony 

1 

few 

manY  #  SELLERS 
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Current  Acquisition  Practices 

Address  Some  Principal-Agent  &  TCE  Concerns 


Multi-Year  Contracting  and  Frequency 

Integrated  Product  and  Process  Development  and  Asset 
Ownership 

Cost  as  an  Independent  Variable  (CAIV)  -  Heightened 
Awareness/Shared  Risk 


Alpha  Contracting  for  ex-ante  Discovery 

Evolutionary  Acquisition  Addresses  Uncertainty  Incrementally 

Single  Process  Initiative  Uses  Frequency  and  Specialization 

Reputation  and  the  Use  of  Past  Performance  Data  and  Award  for 
Best  Value 

Education,  training  and  expansion  of  Acquisition  workforce 
Increased  Monitoring  &  Oversight  (PROGRAM  REVIEWS) 
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Programmatic  Reviews 


•  Multiple,  various  title,  new  and  long-established,  reviews  for 
management  oversight  by  program  executive  officer  (PEO),  Service- 
level  acquisition  executive,  Office  of  the  Secretary  of  Defense  Staff, 
and  Milestone  Decision  Authority/Defense  acquisition  executive 

•  Often  scheduled  and  conducted  according  to  pre-arranged  milestone 
dates  or  per  an  “acquisition  baseline”  established  by  PM  and  multi¬ 
level  staffs  well  in  advance,  or  can  be  on  call/ad  hoc  as  program  issues 
arise 


To  inform  key  staff  members  in  the  Pentagon  (and  Congress)  about 
business  and  technical  progress  of  programs 

Often  result  in  guidance  to  alter  the  course  of  programs  through  re¬ 
structuring,  re-scheduling,  subtracting  funds,  changing  quantities,  etc. 
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Technical  Reviews 


•  Approximately  9  to  1 1  standard  title,  well-established,  and  broadly 
recognized  reviews  within  the  Systems  Engineering  Process  --  More 
technical  than  business  oriented 

•  Conducted  at  the  local  (program  office)  level,  chaired  by  the  PM, 
monitoring  the  technical  progress  of  the  system  prime  contractor  and 
subcontractors 


•  Scheduled  and  conducted  at  the  PM's  discretion  when  appropriate 
(event  or  progress-based  vs.  calendar-based). 


•  Provide  information  to  key  project  members/stakeholders  for  future 
actions.  Facilitate  issue  discovery  and  usually  do  not  change  the 
principal  course  of  a  program 
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NAS  Study  -  PM  Survey 


Main  question:  Can  changes  in  the  number,  content, 
or  sequence  of  program  reviews  help  the  program 
manager  more  successfully  execute  the  program? 

Subjective  answers  from  US  Air  Force  PMs  and  PEOs 
about  reviews  and  their  impact  on  program  execution, 
such  as  time  or  effort  dedicated  to  review  preparation, 
participation,  and  response  to  actions  produced  from 
them 
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Context  of  the  NAS  Study 


•  2000  -  2008,  four  iterations  of  the  DOD  5000  series 
governing  acquisition  programs 

•  Number  and  levels  of  reviews  steadily  increasing 

-  Technical  (Internal  to  PMO  with  Contractor) 

-  Programmatic  (Government  levels  of  oversight) 


•  For  prevention  of  requirements  “creep,”  assurance  of 
adequate  cost  estimation,  and  attainment  of  technical 
maturity  in  regard  to  design  and  producibility 


•  Questions  on  their  cost,  impact,  value... 
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The  Nature  of  Increasing  Reviews 


•  Occur  at  Program  Executive  Office,  Service,  and  Office 
of  Secretary  of  Defense  levels 

•  New  5000  =  New  programmatic  reviews 

-  PSRs,  CSBs,  OTRAs 

-  Extensive  preparatory  activities  for  each 

•  Technical  reviews  that  had  been  discretionary  are  now 
mandatory 
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Implications  of  Theory 


•  T ransaction  Costs 


-  Can  we  use  PM  survey  to  quantify  oversight  and 
monitoring  costs? 


•  Principal-Agent 


-  Is  there  a  difference  in  the  perceived  cost/benefit 
of  oversight  and  monitoring  activities  when  then 
PM  acts  as  the: 


•  Principal  (technical  reviews)? 

•  Agent  (programmatic  reviews)? 
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Starting  Assumptions 


•  Subjective  perceptions  of  relative  cost/benefit  of 
reviews  provide  useful  insights  into  transaction 
costs 

•  Starting  Assumption  -  PMs  are  more  likely  to: 

-  View  technical  reviews  as  less  costly/more  beneficial 

-  View  programmatic  reviews  as  more  costly/less 
beneficial 


Data  Analysis 


•  The  only  data  available  from  the  survey 

report  was  in  summary  form 

-  The  total  response  count  for  each  question  is 
available  but  not  the  individual  responses 


•  Analysis  limited  to  section  2 

(Program  Activity  Overview) 


-  Provided  information  on  pertinent  external 
reviews/reporting  accomplished  by  programs 
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Reviews 


Technical 

Programmatic 

Technology  Readiness  Assessment 
(TRA) 

Defense  Acquisition  Board  (DAB) 
Milestone  Review 

Technology  Maturity  Assessment 
(TMA) 

Defense  Space  Acquisition  Board 
(DSAB)  Milestone  Review 

Manufacturing  Readiness  Review 
(MRR) 

Defense  Acquisition  Board  (DAB)  Status 
Review 

System  Engineering  Assessment  Model 
(SEAM) 

Defense  Space  Acquisition  Board 
(DSAB)  Status  Review 

Overarching  Integrated  Product  Team 
(OIPT)  Review 

Independent  Program  Assessment  (IP A) 

Program  Support  Review  (PSR) 

Logistics  Health  Assessment  (LHA) 

Air  Force  Review  Board  (AFRB) 
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Some  Hypotheses  Tested  (to  date) 


1 .  The  perceived  value  (impact)  of  technical  reviews  is 
higher  than  the  value  of  program  reviews. 

2.  Technical  reviews  are  more  likely  to  be  rated  as  helpful 
(provide  useful  data)  than  program  reviews. 

3.  Technical  reviews  are  less  likely  to  be  perceived  as 
less  beneficial  than  program  reviews. 


4.  Technical  reviews  are  more  likely  to  be  perceived  as 
well  structured  (less  likely  to  be  combined  with  other 
reviews)  than  program  reviews. 

5.  The  perceived  cost  (level  of  documentation  required)  of 
program  reviews  is  significantly  higher  than  the  cost  of 
technical  reviews. 
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Hypothesis  1  -  Results 
(Intuition-Neutral) 


•  Technical  reviews  are  significantly  more 
likely  to  be  seen  as  having  no  impact  on 
program  performance  and  are  somewhat 
less  likely  to  be  seen  as  having  a  negative 
impact  on  program  performance. 


-  Technical  reviews  are  necessary  to  keep 
program  on  track  but  have  little  impact  on  overall 
performance. 

-  Programmatic  reviews  may  result  in  unforeseen 
complications  (PM’s  perspective). 
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Hypothesis  2  -  Results 
(Counterintuitive) 


•  Programmatic  reviews  are  significantly  more 
likely  to  be  seen  as  providing  some  or  lots 
of  useful  data  while  technical  reviews  are 
significantly  more  likely  to  be  seen  as 
providing  little  or  no  useful  data. 


-  A  programmatic  review  may  be  an  opportunity  to 
make  the  PM’s  case  to  senior  leadership. 

-  Technical  issues  may  cover  ground  already 
familiar  to  the  PM. 
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Hypothesis  3  -  Results 
(Counterintuitive) 


•  Technical  reviews  are  more  likely  to  be  rated 
as  least  or  second  least  beneficial 

-  Programmatic  reviews  are  better  opportunities  to 
communicate  with  higher  authorities  (consistent 
with  hypothesis  2  results). 

-  Less  new  information  (for  PMs)  surfaces  in 
technical  reviews,  hence  lower  impact  and  less 
potential  benefit  to  program. 


Hypothesis  4  -  Results 
(Counterintuitive) 


•  Technical  reviews  are  more  likely  to  be 
identified  as  candidates  for  consolidation 

-  Puzzling.  PMs  presumably  have  more  control 
over  structuring  technical  reviews  than 
programmatic. 


Hypothesis  5  -  Results  (Intuitive) 

•  Technical  reviews  are  more  likely  to  be  seen 
as  not  having  enough  documentation. 

-  Summarized  information  provided  to  higher  level 
authorities  is  not  adequate  to  understand 
technical  issues. 

-  Program  managers  understand  the  technical 
details  and  are  aware  of  more  information  that 
could  be  used  to  document  them. 


Some  Preliminary  Conclusions 


•  PMs  find  value  in  programmatic  reviews  even  if 
they  are  more  costly 

-  Essential  to  program  success 

-  Pleased  that  necessary  decisions  have  been  made 

•  Relieved  that  its  over? 


•  Perceived  value  of  technical  reviews  could  be 
lower  because 


-  Delivered  by  contractor  as  required 

•  Not  necessarily  if/when  needed 

-  PM  more  in  tune  with  technical  issues 

•  Review  just  a  formality 
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Continuing  Our  Research 


•  Working  on  getting  more  access  to  survey 
results. 

•  Much  better  analysis  possible  with  more  raw 
data 

-  To  understand  cost/benefit  for  specific  reviews 

-  May  provide  insight  into  effectiveness  of 
incentives  and  principal-agent  relationship 
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Alternative  Options  to  Align  Interests 
of  Agents  with  Principal 


•  When  Measurement  &  Monitoring  is  Costly 

-  Invest  in  more  efficient  oversight  (people  &  processes) 

-  Invest  in  more  complete  contract  (measurement,  etc.) 

-  Cost  Plus  vs.  Fixed  Price  vs.  Incentive  contracts  (NSPS) 

-  Gain-sharing 

-  “Efficiency  wages”  (pay  a  premium  above  market  price) 

-  Tournament  (promotions,  relative  performance, etc.) 

-  Deferred  compensation  (pay  after  experience  is  obtained) 

-  Tenure  (negotiate  promise  of  follow-on  contracts) 

-  Dispute  resolution  mechanisms  (warranties,  etc.) 

-  Invest  in  (build)  identity,  integrity,  trust  &  communication 

-  Invest  in  Transparency  &  Accountability  &  Learning 
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Minimizing  Transaction  Costs 

•  Cut  Coordination  and  Motivation  Costs 


-  Reduce  complexity 

•  Invest  in  more  complete  contract  (search  &  information  costs) 

•  Use  of  Mature  technology 

-  Reduce  uncertainty 

•  Invest  in  more  complete  contract  (bargaining  &  decision  costs) 

•  Measurement,  Monitoring  to  reduce  information  asymmetries 

•  Credible  deterrents  (enforcement,  penalty  clauses,  etc.) 

•  Warranties/Bonding 

-  Increase  frequency 

•  Multi-year  contracts 

•  Reputation 

-  Address  asset  specificity 

•  Reduce  government  requirements  for  specific  investments 

•  Align  Incentives 

•  Government  Owned  Contractor  Operated  (GOCO)  Specific  Assets 

-  Increase  contestability 

•  Government  Standby  Capacity/Expertise 

•  Threat  of  vertical  intearation  (brina  it  back  in-housed 
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Contract  Design 

(Milgrom  &  Roberts  1992) 

•  Informativeness  Principle 

-  Contract  should  include  any  measure  that  reveals  level  of  agent’s  effort 

•  Incentive  Intensity  Principle 

-  Optimal  intensity  of  incentives  depends  on 

•  Opportunity  for  incremental  savings 

•  Ability  to  measure  effort  &  outcomes 

•  Agent’s  tolerance  for  (willingness  to  accept)  risks 

•  Agent’s  responsiveness  to  incentives 

•  Monitoring  Intensity  Principle 

•  Where  optimal  incentive  intensity  is  high,  optimal  monitoring  is  high 

•  Equal  Compensation  Principle 

-  Risk  of  Multi-tasking  (“what  gets  measured  gets  done”) 

•  Target  measures  to  align  Agents  equally  to  all  tasks  valued  by  the  Principal 


Transaction  Costs 


•  TC  costs  include  coordination  and  motivation  costs,  such  as 
search  and  information  costs;  decision,  contracting  and 
incentive  costs;  and  measurement,  monitoring  and 
enforcement  costs 


•  Each  review  is  discrete,  with  a  substantial  workload  impact 
and  cost 


•  For  oversight  (programmatic)  reviews,  possible  diversion  from 
day-to-day  core  program  management  activities 


•  Questionable  value  (GAO  program  outcomes  not  improving 
2000  -  2008) 
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Defense  Acquisition  Transactions 


•  Key  Characteristics  of  Defense  Transactions 

-  Complex 

-  Uncertain 

-  Infrequent  (one-shot) 

-Time  Criticality  (war/peace) 

-  Asset  Specificity  (transaction-specific 
investments) 

-  Limited  Contestability 
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The  TCE  Model 
“Transaction  Costs” 


•  Coordination  Costs 

-  Search  &  Information  Costs 

•  Planning:  Identify  Alternatives 

•  Costs  to  acquire  timely,  accurate  &  relevant  information  (AoA) 

-  Bargaining  &  Decision  Costs 

•  Programming:  Choose  an  Alternative 

•  Costs  of  negotiating  and  writing  contracts 

-  Policing  &  Enforcement  Costs 

•  Budgeting  &  Execution 

•  Costs  of  executing,  monitoring,  measuring,  and  evaluating  contracts 


Motivation  Costs 


-  Costs  of  Countering  Opportunistic  Behavior 

•  Costs  of  encouraging  investments  in  “specific  assets” 
-  Specific  Asset  investments  that  lead  to  “Hold-Up” 
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Key  Characteristics  of  Defense  Acquisition 

Transactions: 


1.  Complexity 

-  Can  we  specify  all  possible  contingencies  in  the  contract? 

•  Credence  or  Experience  goods  (R&D) 

2.  Uncertainty 

-  Imperfect  Information:  probability  distributions? 

•  Demand/Quantity,  Cost,  Schedule,  Performance/Quality 

-  Asymmetric  information:  Does  someone  have  information  advantage? 

3.  Frequency 

-  One-shot  or  repeated  transaction?  Does  reputation  matter? 


4.  Time  Criticality 


-  Wartime  or  peacetime?  How  sensitive  to  price  in  emergencies? 

5.  Asset  Specificity 

-  Can  we  encourage  specific  investments  and  preserve  competition? 
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Transaction-Specific  Asset  Investments 

•  Specific  assets  are  specific  to  the  transaction  & 
lose  much  of  their  value  in  an  alternative  use. 


•  Specific  Assets  come  in  a  variety  of  flavors: 


•  Physical  Assets  (specialized  equipment) 

•  Human  Assets  (special  skills,  knowledge,  training) 

•  Site  Specificity  (special  locations  that  economize  on 
inventory  (EOQ),  transport  costs) 


•  Temporal  Specificity  (investment  in  bottleneck 
activities — PERT/CPM — that  impact  overall  project 
completion  costs  and  schedules) 
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Characteristics  favoring  EXTERNAL  market 

transactions: 

Asset  Specificity. 

If  there  are  few  specific  investments  in  human  or  physical 
capital  required  and  these  investments  retain  much  of 
their  value  outside  the  transaction. 

Frequency : 

If  the  transaction  does  not  require  close  working 
relationships  between  the  parties  involved,  and  take  place 
routinely  (economies  of  scale,  learning  curves). 

Uncertainty/Complexity. 

If  the  transaction  is  standard,  well  understood  so  that 
detailed  agreements  are  unnecessary. 

Performance  Measurement: 

If  the  market  regulates  quality  and  performance. 


How  to  protect  (encourage)  valuable 
transaction-specific  investments? 


•  If  individuals  or  companies  cannot  be  assured  of 
realizing  the  full  value  of  their  specific 
investments,  then  they  will  not  invest  in  specific 
assets  to  achieve  “better,  faster,  cheaper.” 


•  Transactions  that  require  specific  investments 
normally  also  require  credible  protection  against 
early  termination  or  opportunistic  re-negotiation. 


-  Multi-year  contracts  (long-term  contractual  relationship) 

-  Posting  a  bond  (enforceable  termination  commitments) 

-  Reputation 
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